Skip to content

HADOOP-18687. Remove json-smart dependency#5549

Merged
ayushtkn merged 1 commit intoapache:trunkfrom
pjfanning:remove-json-smart
Apr 19, 2023
Merged

HADOOP-18687. Remove json-smart dependency#5549
ayushtkn merged 1 commit intoapache:trunkfrom
pjfanning:remove-json-smart

Conversation

@pjfanning
Copy link
Member

@pjfanning pjfanning commented Apr 12, 2023

complete removal of json-smart jar

@hadoop-yetus
Copy link

hadoop-yetus commented Apr 13, 2023

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 45s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+0 🆗 shelldocs 0m 0s Shelldocs was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+0 🆗 mvndep 16m 13s Maven dependency ordering for branch
+1 💚 mvninstall 26m 4s trunk passed
+1 💚 compile 23m 7s trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚 compile 20m 26s trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚 mvnsite 25m 24s trunk passed
+1 💚 javadoc 8m 2s trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚 javadoc 7m 16s trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚 shadedclient 34m 20s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 30s Maven dependency ordering for patch
+1 💚 mvninstall 22m 0s the patch passed
+1 💚 compile 22m 32s the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚 javac 22m 32s the patch passed
+1 💚 compile 20m 28s the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚 javac 20m 28s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 20m 6s the patch passed
+1 💚 shellcheck 0m 0s No new issues.
+1 💚 javadoc 8m 7s the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚 javadoc 7m 20s the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚 shadedclient 36m 20s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 738m 19s /patch-unit-root.txt root in the patch passed.
+1 💚 asflicense 1m 32s The patch does not generate ASF License warnings.
1011m 34s
Reason Tests
Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner
hadoop.mapreduce.v2.TestMRJobs
hadoop.mapreduce.v2.TestMRJobsWithProfiler
hadoop.mapreduce.v2.TestUberAM
Subsystem Report/Notes
Docker ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5549/1/artifact/out/Dockerfile
GITHUB PR #5549
JIRA Issue HADOOP-18687
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname Linux fca934463446 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / e69ac0b
Default Java Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5549/1/testReport/
Max. process+thread count 3572 (vs. ulimit of 5500)
modules C: hadoop-project . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5549/1/console
versions git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@steveloughran steveloughran changed the title HADOOP-18687: remove json-smart dependency HADOOP-18687. Remove json-smart dependency Apr 14, 2023
@hadoop-yetus
Copy link

hadoop-yetus commented Apr 14, 2023

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 34s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+0 🆗 shelldocs 0m 0s Shelldocs was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+0 🆗 mvndep 16m 12s Maven dependency ordering for branch
+1 💚 mvninstall 25m 30s trunk passed
+1 💚 compile 23m 6s trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚 compile 20m 29s trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚 mvnsite 25m 31s trunk passed
+1 💚 javadoc 8m 2s trunk passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚 javadoc 7m 16s trunk passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚 shadedclient 31m 36s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 31s Maven dependency ordering for patch
+1 💚 mvninstall 22m 13s the patch passed
+1 💚 compile 22m 38s the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚 javac 22m 38s the patch passed
+1 💚 compile 20m 27s the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚 javac 20m 27s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 mvnsite 24m 33s the patch passed
+1 💚 shellcheck 0m 0s No new issues.
+1 💚 javadoc 8m 15s the patch passed with JDK Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1
+1 💚 javadoc 7m 17s the patch passed with JDK Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
+1 💚 shadedclient 36m 19s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 738m 41s /patch-unit-root.txt root in the patch passed.
+1 💚 asflicense 1m 36s The patch does not generate ASF License warnings.
1013m 21s
Reason Tests
Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner
hadoop.mapreduce.v2.TestMRJobs
hadoop.mapreduce.v2.TestMRJobsWithProfiler
hadoop.mapreduce.v2.TestUberAM
Subsystem Report/Notes
Docker ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5549/2/artifact/out/Dockerfile
GITHUB PR #5549
JIRA Issue HADOOP-18687
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname Linux e56c583cf682 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / fc03294
Default Java Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.18+10-post-Ubuntu-0ubuntu120.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_362-8u362-ga-0ubuntu1~20.04.1-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5549/2/testReport/
Max. process+thread count 3203 (vs. ulimit of 5500)
modules C: hadoop-project . U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5549/2/console
versions git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ayushtkn
ayushtkn approved these changes Apr 19, 2023
View reviewed changes
Copy link
Member

@ayushtkn ayushtkn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pjfanning
Copy link
Member Author

pjfanning commented Apr 19, 2023

@ayushtkn I'm not sure yet about this change. Those test failures appear to be consistent issues.

@ayushtkn
Copy link
Member

ayushtkn commented Apr 19, 2023

@pjfanning no worries, I am sure about them, they aren't related. They are failing in the daily build as well. Need to chase them, not getting time unfortunately

The daily build test result. I think all are same as in this PR
https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1199/testReport/

@pjfanning
Copy link
Member Author

pjfanning commented Apr 19, 2023

Thanks @ayushtkn for clarifying. I ran the MR tests locally but they just timed out and I wasn't really sure how to proceed.

@degant
Copy link

degant commented Apr 19, 2023

Do you also plan to upgrade nimbus-jose-jwt to a newer version since the current version shades json-smart 1.3.2 which gets flagged by scanners for both CVE-2021-31684 and CVE-2023-1370?

nimbus-jose-jwt also dropped the json-smart dependency completely with nimbus-jose-jwt 9.24 and replaces it with Gson 2.9.1 (shaded) as seen in the commit history here: https://bitbucket.org/connect2id/nimbus-jose-jwt/commits/tag/9.24

So upgrading nimbus-jose-jwt from 9.8.1 to >9.24.4 could help completely get rid of json-smart from scanners and address HADOOP-18687. Related discussion here: #3299

@pjfanning
Copy link
Member Author

pjfanning commented Apr 19, 2023

Could you raise a separate Jira for nimbus?

@degant
Copy link

degant commented Apr 19, 2023

I don't have an account there and not sure how easy it is to get one

@ayushtkn ayushtkn merged commit b6c0ec7 into apache:trunk Apr 19, 2023
@ayushtkn
Copy link
Member

ayushtkn commented Apr 19, 2023

Merged Thanx @pjfanning for the contribution.

@degant feel free to create a new ticket for nimbus. You can request to create a jira account by filling this form

https://selfserve.apache.org/jira-account.html

@pjfanning pjfanning deleted the remove-json-smart branch April 19, 2023 19:19
@pjfanning
Copy link
Member Author

pjfanning commented Apr 19, 2023

@degant I created https://issues.apache.org/jira/browse/HADOOP-18711

@degant
Copy link

degant commented Apr 19, 2023

Thanks for filing it! I went ahead and requested an account on jira

@ayushtkn
Copy link
Member

ayushtkn commented Apr 20, 2023

For some strange reason, seems the builds have failing on compilation post this
#5575 (comment)
#5483 (comment)

Might not be related, but since it is failing post this only, have reverted this. Would require a rebased PR again or if I find there is something else, can commit it again directly

asfgit pushed a commit that referenced this pull request Apr 20, 2023
@ayushtkn
Revert "HADOOP-18687. Remove json-smart dependency. (#5549). Contribu…
9e3d5c7 
…ted by PJ Fanning."

This reverts commit b6c0ec7.
@ayushtkn
Copy link
Member

ayushtkn commented Apr 20, 2023

I have triggered those builds again post reverting this. @pjfanning can you check once. I think it may be this commit is conflicting with 0d1b4a3556d24641c14bbfc7ae1b985d4a998649, the PR ran build on 0185afafeac26a447b6138b2d74a6f5ed0051d0b which is prior to this one. Just my guess because errors mentioned SBOM

@ayushtkn
Copy link
Member

ayushtkn commented Apr 20, 2023 •
edited
Loading

Context-> #5575 (comment)
Not sure @pjfanning , I triggered those builds again for the HDFS ticket and they failed, the moment I reverted and triggered again it passed. If you say, I can directly re commit the same commit again?

Raised an Infra ticket as well: INFRA-24480

@pjfanning
Copy link
Member Author

pjfanning commented Apr 20, 2023

@ayushtkn can we try again tomorrow perhaps? I have a couple of other PRs that are rebuilding that were affected by a similar issue to this when they first ran.

@ayushtkn
Copy link
Member

ayushtkn commented Apr 20, 2023

Sure, I will commit it tomorrow, Infra confirmed that the issue was on their part only. Sorry, for creating unnecessary noise.

@pjfanning
Copy link
Member Author

pjfanning commented Apr 20, 2023

  • If you're happy it was an infra issue, feel free to reapply this commit whenever suits (today even) - I can rebase the other PRs after this is added back

asfgit pushed a commit that referenced this pull request Apr 20, 2023
@pjfanning @ayushtkn
HADOOP-18687. Remove json-smart dependency. (#5549). Contributed by P…
0918c87 
…J Fanning.

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
@ayushtkn
Copy link
Member

ayushtkn commented Apr 20, 2023

I have pushed again 🤞

rohit-kb pushed a commit to rohit-kb/hadoop that referenced this pull request May 5, 2023
@pjfanning @rohit-kb
HADOOP-18687. Remove json-smart dependency. (apache#5549). Contribute…
b5558ad 
…d by PJ Fanning.

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
steveloughran pushed a commit that referenced this pull request May 9, 2023
@rohit-kb
HADOOP-18687. Remove json-smart dependency. (#5549 + #5524)
771c89a 
Contains 

* HADOOP-18687. hadoop-auth: remove unnecessary dependency on json-smart (#5524)
 Contributed by Michiel de Jong
* HADOOP-18687. Remove json-smart dependency. (#5549).
  Contributed by PJ Fanning.
ferdelyi pushed a commit to ferdelyi/hadoop that referenced this pull request May 26, 2023
@pjfanning @ferdelyi
HADOOP-18687. Remove json-smart dependency. (apache#5549). Contribute…
dc1c889 
…d by PJ Fanning.

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
ferdelyi pushed a commit to ferdelyi/hadoop that referenced this pull request May 26, 2023
@ayushtkn @ferdelyi
Revert "HADOOP-18687. Remove json-smart dependency. (apache#5549). Co…
bb14458 
…ntributed by PJ Fanning."

This reverts commit b6c0ec7.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ayushtkn ayushtkn ayushtkn approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@pjfanning @hadoop-yetus @ayushtkn @degant